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Introduction 


This  presentation  provides  an  overview  of  Software 
Safety  Assurance,  and  includes  a  practical  example 
as  applied  to  the  Vehicle  Management  System 
(VMS)  as  part  of  the  V-22  Engineering 
Manufacturing  Development  (EMD)  Program.  The 
following  topics  are  addressed: 

★  What  is  Software  Safety  Assurance? 

★  Software  Safety  Assurance  Mission 

★  Software  Safety  Assurance  Program 

★  Analysis  Techniques 
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This  presentation  provides  an  overview  of  Software  Safety 
Assurance,  and  includes  a  practical  example  as  applied  to  the 
Vehicle  Management  System  (VMS)  as  part  of  the  V-22 
Engineering  Manufacturing  Development  (EMD)  Program.  The 
following  topics  are  addressed: 

★  What  is  Software  Safety  Assurance? 

★  Software  Safety  Assurance  Mission 

★  Software  Safety  Assurance  Program 

★  Analysis  Techniques 


Introduction  (Continued) 


★  Software  Safety  Engineering  Scope 

★  Software  Safety  Concurrent  Engineering 
Activities 

★  What  are  the  Benefits  of  Software  Safety 
Assurance? 

★  Actions  Taken  To-Date 

★  Summary 
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★  Software  Safety  Engineering  Scope 

★  Software  Safety  Concurrent  Engineering  Activities 

★  What  are  the  Benefits  of  Software  Safety  Assurance? 

★  Actions  Taken  To-Date 

★  Summary 


What  is  Software  Safety  Assurance? 


Software  Safety  Assurance  is  a  process  whereby 
potential  system  hazards,  contributed  to  by  the 
software  or  the  software  environment,  and  their 
causal  factors  are  analyzed  and  proven,  eliminated 
or  mitigated  according  to  their  priority.  If  the 
analysis  fails,  remedy  and  reevaluation  must  be 
done.  Sometimes  residual  risk  must  be  accepted. 
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Software  Safety  Assurance  is  a  process  whereby  potential  system 
hazards,  contributed  to  by  the  software  or  the  software 
environment,  and  their  causal  factors  are  analyzed  and  proven, 
eliminated  or  mitigated  according  to  their  priority.  If  the  analysis 
fails,  remedy  and  reevaluation  must  be  done.  Sometimes  residual 
risk  must  be  accepted. 
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Software  Safety  Assurance  Mission 


To  establish,  help  execute,  and  oversee  a 
Software  Safety  Program  providing  traceable 
software  safety  analyses  evidence  of  hazard 
mitigation  as  input  to  flight  clearance  decisions, 
and  that  the  safety  risk  is  as  low  as  reasonably 
possible. 
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To  establish,  help  execute,  and  oversee  a  Software  Safety  Program 
providing  traceable  software  safety  analyses  evidence  of  hazard 
mitigation  as  input  to  flight  clearance  decisions,  and  that  the  safety 
risk  is  as  low  as  reasonably  possible. 


Software  Safety  Assurance  Program 


The  goal  of  a  Software  Safety  Assurance 
Program  is  to  establish  well  defined  process  tasks 
which  applies  technical  and  administrative 
direction  and  surveillance  through  the  life  cycle 
of  the  project  to  help  prevent  the  loss  of  life, 
property,  or  environment. 

A  Software  Safety  Program  requires  the 
utilization  of  resources  across  the  TEAM. 
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The  goal  of  a  Software  Safety  Assurance  Program  is  to  establish 
well  defined  process  tasks  which  applies  technical  and 
administrative  direction  and  surveillance  through  the  life  cycle  of 
the  project  to  help  prevent  the  loss  of  life,  property,  or  environment. 
A  Software  Safety  Program  requires  the  utilization  of  resources 
across  the  TEAM. 
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Software  Safety  Assurance  Program 


★  Program  Process  Tasks  Include: 

1.  Establish  a  Software  Safety  Working  Group 
consisting  of  the  following  participants: 

♦  Software/Systems  Development  Engineers 

♦  Software/Systems  Safety  Engineers 

♦  Systems  Operators  (i.e.,  pilot) 

♦  Domain  Experts 

♦  Representatives  from  CM,  QA  V&V  and  T&E 
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Establish  a  Software  Safety  Working  Group  consisting  of  the 
following  participants: 

★  Software/Systems  Development  Engineers 
•k  Software/Systems  Safety  Engineers 

k  Systems  Operators  (i.e.,  pilot) 

★  Domain  Experts 

★  Representatives  from  CM,  QA  V&V  and  T&E 
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Software  Safety  Assurance  Program 

★  Program  Process  Tasks  Include  (Cont.): 

2.  Develop  a  Software  Safety  Program  Plan  (SSPP) 

3.  Execute  Prioritized  Functional  Hazard  Analyses 
(FHA).  Some  FHA  activities  include  (but  are  not 
limited  to): 

♦  Develop  a  Preliminary  Hazard  List  (PHL) 

♦  Execute  a  Preliminary  Hazard  Analysis  (PHA) 

♦  Execute  a  Subsystem  Hazard  Analysis  (SSHA) 


DRAFT  10/11/95  Page  7 


Functional  Hazard  Analyses  is  Defined  as  -  The  identification, 
evaluation,  and  management  of  the  system’s  potential  functional 
hazards  that  may  be  contributed  to  by  software  requirements/design 
algorithms,  or  lack  thereof,  (i.e.,  premamre  stores  release), 

PHL  (Preliminary  Hazards  Lists)  consists  of: 

★  Holding  a  brainstorming  session  to  determine  all  potential 
hazards  to  be  analyzed  and, 

★  Working  the  list  for  appropriateness  of  each  potential  hazard. 

PHA  (Preliminary  Hazard  Analysis)  consists  of: 

★  Categorizing  and  Prioritize  the  list  according  to  System 
Safety  MIL-STD-882C  and, 

■A  Having  management  determine  the  list  of  potential  hazards  to 
be  analyzed. 

SSHA  (Subsystem  Hazard  Analysis  consists  of: 

★  Executing  first-level  design  Hazeu'd  Analysis  and, 

★  Execute  detailed  level  design  Hazard  Analysis. 


Software  Safety  Assurance  Program  I 
_ _ (Cont)  I 

★  Program  Process  Tasks  Include  (Cont.): 

4.  Execute  a  Prioritized  Structural  Hazard  Analysis 
(SHA).  Some  software  SHA  activities  include,  but 
are  not  limited  to): 

♦  Product  Integrity  Checklist  Development  and 
Evaluation 

♦  Tool  Integrity  Checklist  Development  and 
Evaluation 

♦  Software  Process  Integrity  Evaluation 

♦  Software  Process  Compliance  Evaluation 

♦  Regression  Analysis 
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Structui  al  Hazard  Analysis  is  Defined  as  -  The  on-going  analysis 
of  development  or  maintenance  processes  and  products  during  the 
entire  life  cycle  with  respect  to  software  engineering  practices/ 
design  features  of  a  system  containing  safety  critical  software,  (i.e., 
no  restricted  op  codes). 

Execute  a  Prioritized  Structural  Hazard  Analysis  (SHA).  Some 
software  SHA  activities  include,  but  are  not  limited  to): 

★  Product  Integrity  Checklist  Development  and  Evaluation 

★  Tool  Integrity  Checklist  Development  and  Evaluation 

★  Software  Process  Integrity  Evaluation 

★  Software  Process  Compliance  Evaluation 

★  Regression  Analysis 


Software  Safety  Assurance  Program 
_ (Cont) _ 


★  Program  Process  Tasks  Include  (Cont.): 

5.  Ensure  Safety  Critical  Requirements  are  traceable 
through  Test. 

6.  Provide  evidence  of  elimination,  mitigation,  or 
accepted  residual  risk. 

♦  Evidence  is  the  documentation  required  to 
substantiate  that  a  hazard  is  proven  eliminated/ 
mitigated.  This  includes  the  analysis  technique 
used  (i.e.,  fault  tree  or  application  of  an  op  code 
tool),  the  actual  analysis,  results  and  remedy  if 
required. 
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If  a  hazard  can  be  reached,  it  must  be  mitigated  and  reevaluated. 

Ensure  Safety  Critical  Requirements  are  traceable  through  Test. 

Provide  evidence  of  elimination,  mitigation,  or  accepted  residual  risk. 

★  Evidence  is  the  documentation  required  to  substantiate  that  a  hazard  is 
proven  eliminated/mitigated.  This  includes  the  analysis  technique  used 
(i.e.,  fault  tree  or  application  of  an  op  code  tool),  the  actual  analysis, 
results  and  remedy  if  required. 

Many  Structural  Hazards  are  evaluated  in  a  good  software  development 
program.  Collecting  the  previously  documented  evidence  may  be  all  that  is 
necessary. 

Many  of  the  elements  of  these  analyses  are  already  considerations  in  the 
software  life  cycle  development.  We  do  not  want  to  duplicate  effort;  however, 
there  is  a  need  to  accumulate  an  audit  trail  of  evidence  that  we  are  at  the  lowest 
possible  risk  and  that  we  cannot  reach  the  hazards  identified. 


The  key  to  the  analysis  concept  is  to  determine  all  of  the  causes  to 
the  hazard  (What  Should  Not  Happen!)  and  compare  that  to  (what 
is  happening).  If  you  can  link  a  path  to  the  root  node  of  the  fault 
tree  you  have  a  problem  that  needs  corrected. 

The  example  identifies  a  possible  algorithm  problem. 

ALGORITHMS  PREVENTING: 

★  Incorrect  Fast  Leak  Detection. 

'k  Loss  of  Local  or  Remote  Swashplates. 

k  Loss  of  Surface  Actuators  must  be  in  place  to  ensure  these 
causes  to  the  loss  of  an  hydraulic  system  are  not  occurring. 


Analysis  Techniques  (Cont.) 


Analysis  Terms  Defined 


?c  Petri  Net  Analysis  (PNA)  -  A  directed  graph  that 
represents  the  logical  states  and  transitions  of  the 
system. 


Event  1 

D^< 

n  .  ^  \i 


(a)  AND  Gate 


(b)  OR  Gate 


D“KDH]'*0^ 

Event  2  And 


0\ 

0*I}*Ch*0 


r  Timed  Petri  Net  -  Petri  Net  that  also  includes 
timing  information. 
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★  Petri  Net  Analysis  (PNA)  is  a  directed  graph  that  represents 
the  logical  states  and  transitions  of  the  system. 

★  Timed  Petri  Net  is  a  Petri  Net  that  also  includes  timing 
information. 

★  These  are  the  parts  of  a  petri  net; 

♦  Places/states 

♦  Transitions 

♦  Directional  flow  arrows 


This  is  a  Petri  net  of  a  candy  machine.  The  dark  dot  signifies  an 
initial  state.  As  the  activity  flows,  the  dot  will  move  throughout  the 
system.  Note  there  isn’t  any  money  in  the  slot,  therefore,  the  state 
is  0  until  the  transition  of  depositing  a  nickel  in  the  machine 
changes  the  state  to  5  cents.  Another  transition  of  depositing  a 
nickel  will  change  the  state  to  10  cents  and  another  to  15  cents. 
Then  a  decision  must  be  made  to  pull  the  lever,  getting  a  15  cent 
candy  bar  and  setting  the  state  back  to  0,  or  depositing  another 
nickel  and  so  on. 
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Analysis  Techniques  (Cont.) 


Analysis  Terms  Defined  (Cont.) 

Software  Fault  Tree  Analysis  (SFTA)  -  A  graphic 
representation  of  parallel  and  sequential 
combinations  of  events  and  system  states  that  can 
lead  to  a  hazard. 

s  Navy’s  Operational  Hazard  Analysis  Linkage 
Technique  (NO  HALT)  -  An  integrated  technique 
that  melds  the  use  of  Petri  Net  representation  of 
system  events  and  the  explicit  fault  representation 
and  diagnosis  in  Fault  Trees  for  a  synergistic 
effect. 
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★  Software  Fault  Tree  Analysis  (SFTA)  is  a  graphic  representation 
of  parallel  and  sequential  combinations  of  events  and  system 
states  that  can  lead  to  a  hazard. 

★  Navy’s  Operational  Hazard  Analysis  Linkage  Technique  (NO 
HALT)  is  an  integrated  technique  that  melds  the  use  of  Petri  Net 
representation  of  system  events  and  the  explicit  fault 
representation  and  diagnosis  in  Fault  Trees  for  a  synergistic 
effect.  I  have  an  entire  brief  describing  this  technique  for 
anyone  that  is  interested. 
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Analysis  Techniques  (Cont.)  I 

PN  &  FT  Selection  Justification 

There  are  several  analysis  techniques  that  may  be 
well  suited  for  a  safety  evaluation  of  a  software 
system,  however: 

Petri  Nets  and  Fault  Trees  are  Mature  Analysis 
Tools. 

?  There  has  been  a  great  deal  of  research  focused 
on  these  two  graphical  representations. 

*  Their  individual  qualities  interleave  well  into  a 
single  effective  analysis  technique. 
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There  are  several  analysis  techniques  that  may  be  well  suited  for  a 
safety  evaluation  of  a  software  system,  however: 

★  Petri  Nets  and  Fault  Trees  are  Mature  Analysis  Tools. 

★  There  has  been  a  great  deal  of  research  focused  on  these  two 
graphical  representations. 

★  Their  individual  qualities  interleave  well  into  a  single 
effective  analysis  technique. 


This  is  the  linking  of  a  fault  tree  to  a  Petri  net  in  an  academic 
example  of  a  traffic  light  sequence.  We  should  all  be  familiar  with 
this.  It  was  obvious  that  there  was  no  second  delay  going  from  red 
to  green  on  the  east/west  traffic  light  when  we  applied  this 
technique.  Again,  when  you  can  reach  the  root  of  the  hazard 
through  this  linkage,  you  have  a  problem. 


Here  is  the  rigorous  semantic  model  that  reflects  this  linkage  for  the 
mathematicians.  Even  though  graphics  is  the  preferred 
representation,  this  textual  representation  will  give  an  equivalency. 
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Here  are  the  constraints. 
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Partial  V-22  VMS  Potential  Functional  Hazard  List 

1 .  Simultaneous  Fault  Restart:  Loss  of  Channel  ID 

t 

2.  Swashplate  Actuator  Overtilts 

3.  Loss  of  Hydraulic  System 

4.  Inadvertent  Engine  Shutdown 

5.  Impact  of  SLL  on  Aircraft  Safety 

-  Excess  Flapping 

-  Commanded  Loads  Exceed  Design  Limits 

6.  Driving  Both  Ends  of  Ball  Screw  Simultaneously 
Results  in  Loss  of  Actuator  Control 
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Here  are  a  few  of  the  potential  functional  hazards  that  were 
prioritized  and  categorized  to  be  the  most  critical  to  analyze: 

1.  Simultaneous  Fault  Restart:  Loss  of  Channel  ID 

2.  Swashplate  Actuator  Overtilts 

3.  Loss  of  Hydraulic  System 

4.  Inadvertent  Engine  Shutdown 

5.  Impact  of  SLL  on  Aircraft  Safety 

-  Excess  Flapping 

-  Commanded  Loads  Exceed  Design  Limits 

6.  Driving  Both  Ends  of  Ball  Screw  Simultaneously  Results  in 
Loss  of  Actuator  Control 


Analysis  Techniques  (Cont.) 


V-22  VMS  Fault  Tree  -  Partial  Example 


Here  is  a  portion  of  the  fault  tree  addressing  the  loss  of  hydraulics. 
Remember  this  page  is  the  format-how  we  can  take  the  analysis  to 
additional  pages.  This  was  the  first  page. 


This  portion  shows  that  you  can  go  all  the  way  to  the  algorithm 
level.  Note  that  you  can  check  all  the  pieces  of  the  algorithm  as 
well  as  the  frequency  at  which  it  executes. 


This  portion  shows  where  we  connected  the  fault  tree  to 
documentation.  We  were  unable  to  graph  out  the  associated  Petri 
nets  due  to  time  constraints,  but  we  are  able  to  progressively 
connect  them  to  the  appropriate  documentation,  specifications, 
requirements,  design,  and  even  code. 


21 


This  final  example  graph  shows  that  we  can  easily  represent 
hardware,  human  factors,  and  software  nodes  and  their 
interoperability  within  the  causal  factor  nodes. 


Analysis  Techniques  (Cont.) 

V-22  VMS  Potential  Structural  Hazards 

•k  Identification  of  safety  critical  CSU’s,  CSCs, 
CSCI 

★  Assembly  Language 

♦  Improper  data  typing 

♦  Stack  overflow 

♦  Improper  scaling 

♦  Illegal  opcodes 

★  Lack  of  tool  validation 
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Some  of  the  structural  hazards  identified  and  addressed  in  the  V-22 
safety  program  are: 

★  Identification  of  safety  critical  CSU’s,  CSCs,  CSCI 

★  Assembly  Language 

♦  Improper  data  typing 

♦  Stack  overflow 

♦  Improper  scaling 

♦  Illegal  opcodes 

★ 


Lack  of  tool  validation 


Analysis  Techniques  (Cont.) 

V-22  VMS  Potential  Structural  Hazards  (Cont.) 

★  Non-compliance  with  configuration  management 
process 

★  Error  in  safe  state  return 

★  Inadvertent  jumps 

★  Recursive  loops 

★  Safety  kernel  failure 

★  Inadequate  regression  testing 
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More  potential  structural  hazards  are  : 

★  Non-compliance  with  configuration  management  process 

★  Error  in  safe  state  return 

★  Inadvertent  jumps 

★  Recursive  loops 

Tfer  Safety  kernel  failure 

★  Inadequate  regression  testing 


Analysis  Techniques  (Cont) 


y-22  VMS  Hazard  Analysis  Results  To-Date 


★  Team  Focus 

★  Influence  Design  Decisions 
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We  really  have  executed  the  team  concept  within  our  safety 
program.  There  has  been  some  reluctance;  however,  when  they 
understand  the  goal,  there  has  been  good  participation.  All  of  the 
domains  have  been  exercised.  This  has  been  a  merging  of  the 
software  system  engineers  and  the  safety  engineers.  It  takes  both. 
Neither  can  execute  software  safety  assurance  alone. 


We  have  been  able  to  influence  the  design  at  the  most  advantageous 
time,  the  beginning.  While  gathering  data  for  the  hydraulic  hazard, 
systems  engineers  found  a  problem  and  were  able  to  correct  it  in 
the  pen  and  ink  phase.  There  could  be  no  more  ideal  time. 


Software  Safety  Engineering  Activities  span  across  systems  and 
software  engineering,  exercising  hazard  analysis  and  safety  critical 
specification  traceability  to  test. 

The  disciplines  merge  when  going  from  system  design  to  software 
design. 


Usually  the  safety  engineer  determines  the  general  (first  level) 
hardware,  human  factors,  and  software  causes  to  the  root  hazard. 
The  software  systems  designers  can  then  assist  his  analysis  by 
taking  the  analysis  into  the  software  design.  They  can  also 
participate  in  the  safety  program  by  executing  the  tagging  of  safety 
critical  requirements  and  tracing  to  test,  process  and  product 
progressive  evaluation,  and  certification  portions  of  the  software 
safety  assurance  process. 
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What  are  the  Benefits  of  Software  Safety 
Assurance? 


★  Provides  evidence  of  an  auditable  trail  that 
potential  hazards  have  been  analyzed  and  proven 
eliminated  or  within  reasonable  risk. 

★  Uncovers  specification  oversights.  Many  known 
accidents  have  occurred  because  there  was  no 
consideration  of  a  particular  hazardous  situation. 

★  Provides  analysis  results  that  may  double  as 
design  considerations. 
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Some  benefits  of  structured  software  safety  assurance  are: 

-k  It  provides  evidence  of  an  auditable  trail  that  potential  hazards 
have  been  analyzed  and  proven  eliminated  or  within  reasonable 
risk. 

★  It  uncovers  specification  oversights.  Many  known  accidents 
have  occurred  because  there  was  no  consideration  of  a  particular 
hazardous  situation. 

★  Provides  analysis  results  that  may  double  as  design 
considerations. 


What  are  the  Benefits  of  Software  Safety 
Assurance  ?  ( Cont ) 

★  Determines  what  hazard  contributing  factors 
should  not  occur  and  then  ensures  they  don’t. 
Significant  cost  savings  can  be  attributed  if  these 
problems  are  resolved  early. 

★  Pictorially  represents  the  evaluation  of  each 
functional  hazard  so  management  and  auditing 
groups,  as  well  as  technical  experts,  can  easily 
understand  the  results. 

★  Provides  clear,  repeatable  reliable  methods  for 
software  safety  analysis. 
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Also  it  helps  DESIGN  OUT  HAZARDS  because: 

★  It  determines  what  hazard  contributing  factors  should  not  occur 
and  then  ensures  they  don’t.  Significant  cost  savings  can  be 
attributed  if  these  problems  are  resolved  early. 

★  It  pictorially  represents  the  evaluation  of  each  functional  hazard 
so  management  and  auditing  groups,  as  well  as  technical 
experts,  can  easily  understand  the  results. 

★  Provides  clear,  repeatable  reliable  methods  for  software  safety 
analysis. 


What  are  the  Benefits  of  Software  Safety 
_ Assurance?  (Cont) _ 


★  May  identify  latent  software  faults  or  reactions  to 
certain  scenarios  not  usually  found  until  testing,  or 
even  worse,  until  the  prototype  is  out  in  the  field. 

BOTTOM  LINE  .  Cost  Avoidance  far 

outweighs  the  cost  for  the  additional  Software 
Safety  Program  activities  needed  to  ensure  the 
software/software  environment  is  at  the  lowest 
possible  safety  risk. 
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ALSO: 

■k  It  may  identify  latent  software  faults  or  reactions  to  certain 
scenarios  not  usually  found  until  testing,  or  even  worse,  until  the 
prototype  is  out  in  the  field. 

The  BOTTOM  LINE  is  that  Cost  Avoidance  far  outweighs  the 
cost  for  the  additional  Software  Safety  Program  activities 
needed  to  ensure  the  software/software  environment  is  at  the 
lowest  possible  safety  risk. 


Actions  Taken  to  Date 


★  IEEE,  FAA,  NASA,  ISO,  have  established  limited 
software  safety  standards  and  direction. 

★  A  Joint  Software  System  Safety  group,  of  which 
NAVAIR  participates,  has  been  chartered  to 
develop  a  Software  Safety  Handbook  (March 
1996). 

★  NAVAIR  is  also  participating  in  developing 
international  standards. 
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Some  activities  contributing  to  standardization  are: 

★  IEEE,  FAA,  NASA,  ISO,  have  established  limited  software 
safety  standards  and  direction. 

★  A  Joint  Software  System  Safety  group,  of  which  NAVAIR 
participates,  has  been  chartered  to  develop  a  Software  Safety 
Handbook  (March  1996). 

★  NAVAIR  is  also  participating  in  developing  international 
standards. 


Summary 


•k  Safety  cannot  be  guaranteed;  however,  risk  to  life, 
property,  environment  and  cost  can  be 
significantly  reduced  by  establishing  a  Software 
Safety  Program  that  provides  traceable  software 
safety  analyses  evidence  of  hazard  mitigation  as 
input  to  flight  clearance  decisions,  and  that  the 
safety  risk  is  as  low  as  reasonable  possible. 

★  Integrated  analysis  methods  should  enhance  early 
fault  discovery  by  focusing  on  the  key  safety- 
critical  portions  of  the  software  and  avoiding 
redundant  analysis. 
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IN  SUMMARY: 

■k  Safety  cannot  be  guaranteed;  however,  risk  to  life,  property, 
environment  and  cost  can  be  significantly  reduced  by 
establishing  a  Software  Safety  Program  that  provides  traceable 
software  safety  analyses  evidence  of  hazard  mitigation  as  input 
to  flight  clearance  decisions,  and  that  the  safety  risk  is  as  low  as 
reasonable  possible. 

★  Integrated  analysis  methods  should  enhance  early  fault 
discovery  by  focusing  on  the  key  safety-critical  portions  of  the 
software  and  avoiding  redundant  analysis. 


★  The  analysis  techniques  identified  in  this 
presentation  have  been  successfully  applied  to  the 
VMS  as  part  of  the  V-22  EMD  Program.  By  using 
these  techniques,  potential  system/software 
hazards  are  being  analyzed  and  proven  eliminated 
or  mitigated  according  to  their  priority. 
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AND  FINALLY: 

★  The  analysis  techniques  identified  in  this  presentation  have  been 
successfully  applied  to  the  VMS  as  part  of  the  V-22  EMD 
Program.  By  using  these  techniques,  potential  system/software 
hazards  are  being  analyzed  and  proven  eliminated  or  mitigated 
according  to  their  priority. 


PROVIDING  EVIDENCE!!! 


